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The Center for Substance Abuse Prevention (CSAP) in the Substance Abuse and Mental Health Services Administration 
(SAMHSA) is the Nations lead agency for substance abuse prevention. The Center funds community-based organi 2 ations, 
umversities, behavioral health providers, and public agencies to identify effective prevention programs and practices and dis- 
seminates findings, program models, and other prevention materials to practitioners and policymakers across the country. 
This document summarizes the findings of CSAP’s National Cross-Site Evaluation of High-Risk Youth Programs. This 
large multiple-site evaluation was designed to assess 48 prevention programs and to identify those program characteristics 
that are associated with strong substance abuse prevention outcomes. The rigor and relevance of The National Cross-Site 
Evaluation earned CSAP the 2000 Outstanding Study of the Year Award from the American Evaluation Association. 

The National High-Risk Youth Demonstration (fiinded from 1987 to 1995) has been one of the most ambitious and produc- 
tive of CSAP’s funding initiatives. In its early years, the demonstration focused on identifying promising approaches to pre- 
vention. As the demonstration matured, individual site and cross-site research produced knowledge about risk and protective 
factors related to substance use and helped to identify model programs for effective prevention. Research and experience in 
the High-Risk Youth Demonstration also contributed to the awareness of the importance of culturally sensitive, age-appro- 
priate, and gender-specific programming. This progress in prevention theory and practice laid the foundation for the 
research reported in this document, the largest and most comprehensive of CSAP’s Hi gh -Risk Youth studies. 

In addition to individual-level information on substance use, risk, and protection, the evaluation includes detailed informa- 
tion on the nature and amount of prevention services in which each child participated, as well as systematic process informa- 
tion on the study programs. This monograph highlights the study’s contributions to the growing evidence that “Prevention 
Works. More specifically, this document summarizes major findings concerning the effectiveness of prevention programs in 
reducing rates of substance use among youth at high risk. The analyses identify important characteristics of program setting 
(e.g., implementing organization, and the setting in which the program is delivered) that condition program effectiveness, and 
the approaches to program content and delivery method that have contributed most to effective progra mmin g in these real- 
world prevention programs. 

This document is part of a series of Points of Prevention publications that document the contribution of CSAP’s National 
Cross-Site Evaluation of High-Risk Youth Programs to prevention knowledge and provide science-based guidance for 
improved prevention policy and practice. 



Charles G. Curie, M.A., A.C.S.W. 
Administrator 

Substance Abuse and Mental Health 
Services Administration 



Ruth Sanchez- Way, Ph.D. 

Director 

Center for Substance Abuse Prevention 
Substance Abuse and Mental Health 
Services Administration 
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In the last decade, prevention of substance abuse has become a standard part of the awareness and practice of teachers, 
youth workers, and community-based organkations. Beginning with the visible outbreak of illicit drug use among youth in 
the 1960’s, awareness of the risk, harm, and reduction of opportunity associated with alcohol, tobacco, and drug use among 
young people has increased dramatically. Public agencies and private foundations have supported research that has produced 
important insights into the initiation and prevalence of substance use among youth and into its association with poor educa- 
tional, behavioral, and social outcomes. Practitioners and evaluators have developed and tested prevention strategies and pro- 
grams that contribute to positive youth development and reduce substance use. Social institutions and private citizens have 
made the prevention of substance use an integral part of efforts to support and promote positive future outcomes for 
America’s youth. Progress in providing prevention is evident. 

As concerned professionals and citizens have recognized the need for prevention programming, they have also become more 
concerned about how to make their prevention activities as effective as possible. Many of their questions focus on the design 
of program content and delivery. What types of learning experiences or activities are most effective in preventing substance 
use? What are the most effective ways to deliver these activities? Are different organizational arrangements more or less 
effective in providing prevention services to youth? 

Scientific study is important to answering these questions and developing guides to effective programming. However, preven- 
tion professionals and volunteers also need to know how to make effective prevention practices work in real community set- 
tings, typically with limited resources. Designing and implementing effective prevention programs depends on science-based 
knowledge tempered and shaped by experience and success in actual community settings. The bulk of prevention research 
has concentrated on school-based programs for the general youth population (Paglia and Room, 1999). Relatively little infor- 
mation is available to prevention practitioners about effective programming for high-risk youth. 

This monograph is one in the Points of Prevention series, which reports findings from a landmark evaluation that is unique- 
ly suited to provide information on the design and implementation of prevention programs for high-risk youth in actual 
community conditions. The National Cross-Site Evaluation of High-Risk Youth Programs is a large study by the Center for 
Substance Abuse Prevention (CSAP) within the Substance Abuse and Mental Health Services Admimstration (SAMHSA). 
The study includes 48 programs originally funded by CSAP in 1994 or 1995 to implement prevention programs for youth at 
high risk for substance abuse. These programs were not selected as model programs, nor were they implemented specifically 
for the purposes of the study. The findings for these programs are not the product of successful studies already published. 
These are actual programs implemented by a variety of organizations with different resources, staffing, and experience. The 
programs were implemented in communities at high risk and represent the experience and challenges of programs in these 
conditions. Not all of the programs succeeded, and of those that did achieve their objectives, some were much more suc- 
cessful than others. In short, this study provides the opportunity to learn about a range of program experience in actual 
community conditions, and to learn what design and implementation features contribute to effectiveness in reaching preven- 
tion objectives. 
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Designing and Implementing Effective Prevention Programs examines the experience of 48 programs providing prevention 
services to youth at high risk for substance abuse in diverse communities across the country The monograph presents infor- 
mation and findings on several major issues that prevention workers must consider in making their programs effective. 



O The effectiveness of prevention programs depends on decisions about the setting in which they are delivered, the design 
of program content and delivery, and the ways in which the program is implemented. This monograph identifies major 
considerations concerning setting, design, and implementation and provides guidance in thinking about how they may 
interact to make prevention efforts more or less effective. 

O The National Cross-Site Evaluation provides information comparing the effectiveness of the 48 study programs in 
achieving outcome objectives. It also includes detailed information comparing these programs on their strategies and 
activities. This information is used to provide science-based answers to questions about what makes prevention pro- 
grams effective for youth at high risk. 

O The most important product of prevention studies is to provide practical information that designers and implementers 
of prevention efforts can use to benefit the youth they serve. This monograph concludes with guides and considerations 
for strengthening prevention programs and activities. 



The following section briefly introduces CSAP s National Cross -Site Evaluation, focusing on the way in which program 
effectiveness is measured, and the ways in which comparable information on program design and implementation was gath- 
ered. Subsequent sections present major findings and lessons concerning how to design and implement effective prevention 
programs for high-risk youth. 
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The National Cross-Site Evaluation of High-Risk Youth Programs has produced a unique collection of information on pre- 
vention programs, the youth who participate in them, and the effects of this participation on substance use and factors that 
protect youth from substance use. The study includes extensive questionnaire information collected from approximately 
6,000 youth who participated in prevention programs funded by CSAP, and approximately 4,500 similar comparison group 
youth who did not participate in the CSAP programs. Data were collected from these youth at four points in time, including 
program entry, program exit, 6 months after exit, and 1 8 months after exit. This large data set on individual youth is the 
focal point of analyses presented in other monographs within the Points of Prevention series. 

The National Cross-Site Evaluation also produced extensive and detailed information about the organi 2 ation of the 48 pro- 
grams in the study sample, and on the services they provide to youth. This program-level information, plus information 
about the effectiveness of study programs in achieving intended outcomes, provides the basis for the findings and recom- 
mendations on how to design and implement effective prevention programs that are presented in this monograph. 

Program DescripHons and Measnras 

Accurate and detailed information on program organkation and content was critical to achieving the National Cross-Site 
Evaluation’s full potential for contributing to knowledge about what works in prevention programming. In single-site evalua- 
tions, this program-level information is part of the description of the program being studied. In a multisite study such as the 
National Cross-Site Evaluation, these measures are potentially important explanations of differences in program effective- 
ness. The analyst can ask whether certain characteristics of programs are associated with greater effectiveness in achieving 
intended outcomes. For this study, program-level measures were constructed from systematic data collected and organized in 
the following ways. 

Site Descriptions. The National Cross-Site Evaluation worked closely with each of the 48 program sites to clearly describe 
their organization, management, and services in comparable ways. Program managers and staff, as well as representatives of 
the organizations that administered or collaborated with the programs, were interviewed extensively in 2- to 3-day site visits 
to each site, and in followup interviews by telephone. Program documents such as proposals for funding, lesson plans, and 
curricula were studied, and program activities were observed when possible. The detailed information from these site visits 
was used to code more than 400 closed-ended variables profiling the community and organizational environments of pro- 
grams, management and staffing, program objectives and prevention strategies, planned program services, and participant 
recruitment. Narrative descriptions, elaborations, and explanations were added to the coded site profiles to capture the 
unique circumstances and context of each program.^ To provide an overview of each program site, brief case descriptions 
were prepared using a common oudine. 

Service Profiles. The National Cross-Site Evaluation collected information of unprecedented detail and completeness con- 
cerning the services received by youth participating in each program site. The process of conceptualizing and collecting these 
“dosage” data was complex and careful. Working from program proposals and preliminary site visits, six categories of pro- 
gram content (prevention strategy), and four categories of service delivery method were developed by the evaluation team. 
Utilizing these coding categories, detailed “dosage coding plans” were developed for each program. These coding plans 
mapped all planned program activities so that each activity was assigned to a prevention strategy and a delivery method cate- 
gory. Intervention activities were organized so that several different interventions could be coded within a program session, 
allowing detailed documentation of the amount and type of contact each participant youth had with their program. Contact 
information was coded for more than 217,000 interventions across the 48 programs. 

9 



^ For a complete description of the site visit protocol and the resulting database, see the National Cross-Site Evaluation of High Risk Youth Programs Final Technical 
Report, available through the Center for Substance Abuse Prevention. 
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These data provide rich and detailed information that was used to develop the program and site characteristics analyzed 
below. It is this program-level data that provides the unique content of this monograph. They allow us to assess the ways in 
which differences in program setting, design and implementation affect the achievement of program objectives. The foUow- 
mg section reviews the outcome measures developed to assess the degree to which the programs achieve their major objec- 
tives. 



Measures of Program EffecHveoess 

The National Cross-Site Evaluation provides an unusual opportunity to assess the effectiveness of a broad range of pro- 
grams implemented under real community conditions. The cross-site programs were not selected because they were model 
programs. The programs in the study represent the day-to-day challenges of design and implementation confronted by pre- 
ventdon workers in the field. 

Substance Use. While many of these programs had specific objectives that were relatively unique to their design or their 
target population, they aU shared a major objective to reduce the levels of substance use by participating youth as compared 
to comparison youth who are similar in circumstances, demographics, attitudes, and behaviors, but did not participate in a 
CSAP prevention program. To assess the effectiveness of the programs in achieving this overall objective, four measures of 
substance use are employed in the following analyses as program outcomes. 

O Thirty-day Cigarette Use. While some programs specifically addressed tobacco, particularly because of its health conse- 
quences, it was not as prominently targeted as the more psycho-active and intoxicating drugs. Moreover, cigarette pre- 
vention IS a well-developed field independent of alcohol, marijuana, and illicit drug prevention. Many CSAP programs 
were not part of the tobacco health network and rarely used focused tobacco curricula. Of the three drugs measured as 
outcomes, cigarette use received the least focused prevention attention, and lesser effects on smoking behavior may be 
expected. 

O Thtrty-day jAlcohol Use. Alcohol use was the most widespread focal topic of substance use prevention in cross-site pro- 
grams. Alcohol is used more easily and more prevdently than any drug in the study. It may be expected that alcohol use 
would be positively impacted by more cross-site prevention programs than other substances. 

O Thirty-day Marijuana Use. Marijuana use initiates at a later age than alcohol but rises extremely rapidly, particularly among 
males. Marijuana use is becoming a major concern in prevention as its use proliferates, but its development and patterns 
of use are less well understood than for alcohol. 

O Thirty-day Substance Use. This measure combines use of cigarettes, alcohol, and marijuana to assess overall change in ado- 
lescent substance use for the three most widely used substances in this population. 
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Substance Use Attitudes. While reductions in substance use are the ultimate objectives of the CSAP programs, changes in 
other factors that are thought to be related to substance use are common intermediate outcomes. Accordingly, many of the 
study programs were designed to change the attitudes held by youth regarding substance use. The Cross-Site Evaluation cap- 
tured data on this traditional prevention program objective. 



O Negative Attitudes Toward Substance Use. Intervention strategies were frequently used to increase awareness of the negative 
consequences of alcohol, tobacco, and illicit drug (ATD) use and to promote negative attitudes about substance use. 

A standard scale was used to measure negative attitudes toward substance use among the study youth. 

Risk and Protection Factors. In response to the CSAP solicitation for funding proposals, each of the programs in this 
study was designed with recognition of the syndrome of related circumstances and behaviors in youth referred to as the risk 
and protection framework. U nlike the common focus on reducing substance use, the programs varied substantially in the 
specific definition and targeting of the risk and protective factors they intend to impact. It follows that there may be less 
commonality across programs in impacting risk and protection than in impacting substance use. 

Nevertheless, the analysis presented here assesses the degree to which study programs impacted two important protective 

factors school connectedness and family connectedness. These variables were selected because analyses of the cross-site 

data that are reported elsewhere^ identify these major protective factors as central to creating, supporting, and sustaining 
opportunities for positive development. These variables, representing the development of positive and rewarding connec- 
tions to important social environments, were shown to be highly related to other protective factors, and to substance use. 

One or the other of these major variables was often the direct or indirect target of the 48 programs in the study sites. 
Because those programs focused their attention on working directly with youth rather than with environmental influences, 
the primary impacts of the programs on connectedness would come through building more positive orientations to 
environmental influences (e.g., family and school), and through improving their competencies in behaving in a meaningful 
and fulfilling manner in these environments. 

O School Connectedness. This protective factor is central for linking youth to a positive social setting that can offer many 
opportunities for involvement and reward. Many of the programs funded by CSAP worked closely with schools and 
many offered targeted skills or educational support programming that worked directly on the youth s effort and success 
related to school. School connectedness is an important protective factor that is closely related to the activities of many 
of the programs in the study. The cross-site programs should show widespread effects on school connectedness. 

O Family Connectedness. A second important protective factor in this analysis is family connectedness. Few of the cross-site 
programs spent significant program time in activities that included family members in an intensive way. Fewer programs 
focused on strength enin g behaviors such as family bonding and family communication by working with youth and family 
together. Consequently, it is not expected that these programs, which focused primarily on the youth themselves, would 
impact family connectedness as strongly as school connectedness. 
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Ppogpam Effects 

The fundamental question asked in evaluation is, “Did the program work — -did it attain its objectives?” For policymakers 
and prevention practitioners, the important answers are not about a single program but about many programs in a variety 
of commumties, offered by different organi2ations, and serving different types of people. Policymakers and practitioners 
know, from the Uterature on evaluation and from experience, that some programs do attain their objectives and some do 
not They are less certain about the portion of existing programs that succeed or why some programs do better than others. 
This section presents information that will help reduce this uncertainty and provide the type of uniform information over 
multiple programs that decision makers need. 



In the National Cross-Site Evaluation, 48 programs provided services designed to reduce substance use and strengthen 
internal protective factors in the participating youth. Past research and experience indicates that these programs would 
experience a range of success in their programming — that some would be more successful than others. By deterrnining 
a statistical effect sEe for each program, we can identify the percentage of study programs that are effective with respect 
to a particular program outcome or combination of outcomes.^ 



As shown in Figure 1, the proportion of programs that achieved success varies on the different outcome measures. An 
effect size of 0.20 or above is used to determine success."*^ The length of the bars indicates the percentage of programs 
with positive effect sizes of 0.20 or above. 



Figure 1 

Percent of Programs with Meaningful Positive Effects 
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The distributions in Figure 1 support several findings. 



O If all of the outcome measures are considered, 60 percent of the programs produced a meaningfiil positive result in at 
least one outcome area. 



O Among all of the individual outcomes, the programs were able to effect substance use attitudes the most. Over one-third 
(36%) of the programs produced a meaningful increase in negative attitudes toward substance use. The relationship that 
attitude change has with actual reductions in substance use will be explored later in this monograph. 

O Reductions in alcohol use represent the highest percentage of programs attaining a meaningfiil impact on substance use. 

Nearly one-fourth (24%) of the programs reduced alcohol use. This finding reflects the most prevalent substance use 
emphasis of the sample programs. 



3 As used here, effect sizes are simply the difference between participant group change and comparison group change on the outcome measure for each program 
sandMchzed to make it comp^ble across the different programs. For a complete discussion, see the National Cross-Site Evaluation of High Risk Youth ProgrLns 
rinal Technical Report, available through the Center for Substance Abuse Prevention. ° 

* utilize *e fuU r^ge in effect sizes across programs, the standard of .20 to .49 constituting a smaU effect (but one attributable to 

program).^50 to .79 consnmMg moderate effects, and .80 and above constituting a large effect is adopted as cutpoint criterion of effectiveness. The typical experienc 

of prevention research, and the restneted vanance m the substance use measures in this study (a large portion of non-users, particularly in the younger ages) justiries th 
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O School connectedness is the specific outcome in which programs overall had the third highest level of success. 

Approximately 17 percent of the programs produced meaningful positive effects on this pivotal protective factor. This 
finding is consistent with the greater emphasis that the programs gave to connecting youth to their schools as compared 
to connecting youth to other important social environments. 

O Approximately 15 percent of the programs produced meaningful positive effects on marijuana use. Marijuana use 

demonstrated much less variation in effects than alcohol, partly reflecting the very low levels of use in many programs. 

O Despite the very low levels of actual program time working with famihes in most programs, 13 percent of the programs 
produced a meaningful positive effect on family connectedness. 

O Of the six major outcomes, the programs had the lowest level of program success in producing meaningful reduction in 
expected rates of cigarette use. Fewer than 1 in 10 (9%) of the programs attained a meaningful positive effect. This is 
consistent with the programs’ targeted objectives as identified above. 



In review, the distribution of findings on outcomes for youth while they are participating in their respective programs offers 
positive results. More than one-third (35%) of the programs produce a meaningful positive effect on one of the three sub- 
stance use outcomes. If the risk and protective factors and attitudes are considered, almost two-thirds (60%) of the pro- 
grams produce a meaningful positive result in at least one major outcome area. 

The Consistency of Program Effects 

The findings presented above indicate that programs frequently may have meaningful positive effects in one or a few out- 
come areas, but not in others. While this pattern suggests that meaningful outcomes may reflect particular program 
emphases in one area and relative lack of emphasis in others, it does not suggest that success in one area does not tend to go 
with success in another. For example, although every program that achieves a meaningful change in attitudes toward sub- 
stance use may not achieve a meaningful change in the effect size for substance use itself, it may be that there is a tendency 
for use patterns to move in a corresponding positive direction when attitudes change. 

The relationships among effect sizes of the six outcome variables are displayed in Table 1. Higher numbers indicate a 
stronger tendency for the effect sizes to move together. 



Table 1 

Relationship of Baseline-to-Exit Effect Sizes 
(n = 46) 





Alcohol 

Uie 


MAriJoana 

Uie 


Family 

Coonectedness 


School 

Connectedacu 


Substance Use 
Attitudes 


cigarette Use 


0.330* 


0.012 


0.069 


0.244 


0.086 


Alcohol Use 




0.389** 


0.221* 


0.497** 


0.052 


Maiijaana Use 






0.280* 


0.200 


0.122 


Family 

Connectedness 








•• 

0.571 


-0.063 


School 

Connectedness 










-0.097 



Note; Effect size variables are non-normally distributed, therefore the 
Spear man*s rho correlation coefficient was used. 

♦Correlation is significant at the 0.05 level (1-tailed). **Correlation is significant at the 0.01 level (1 -tailed). 
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The pattern of association among program effects on the different outcomes supports several important observations. 



O Even though most programs do not achieve meaningful effects on more than one substance (usually alcohol), the effect 
sizes for alcohol use with cigarette and marijuana use are interrelated at moderate to strong levels. In other words, when 
programs affect alcohol use positively, they also tend to have positive effects (even if of lesser magnitude) on the use of 
other substances. 

O Cigarette use effects are not strongly related to marijuana use effects. Thus, when programs reduce cigarette use, they do 
not necessarily tend to reduce marijuana use. 

O Program effects on school connectedness are strongly related to alcohol effects, and to a lesser extent to effects on mari< 
juana and cigarettes. Family connectedness effects are moderately connected to alcohol and marijuana use. This pattern 
supports the expectation that impacting risk and protective factors and, in this case, improving connectedness to impor- 
tant social settings will go with less risk for substance use. 

O By contrast, program effects on substance use attitudes are virtually unrelated to effects for alcohol or cigarettes, are only 
weakly related to effects on marijuana use, and have very weak negative relationships with school and family connected- 
ness. None of these relationships is statistically significant. 



The inconsistency in effect sizes for substance use attitudes with other outcomes has important implications. First, it again 
highlights the potential fallacy of assuming that behavior follows expressed attitudes. Clearly, professed attitudes changed in 
many programs without accompanying changes in behavior. Second, these findings suggest that time devoted to activities 
that target change in attitudes about substance use would be better spent on activities designed to impact risk and protective 
factors, particularly those related to building the connectedness of youth with meaningful social environments. For this rea- 
son, program effects on substance use attitudes are not included in subsequent analyses in this monograph. 

The following section introduces a simple organizational framework for thinking about the program setting, design, and 
implementation factors that shape programs and may have consequences for program effectiveness. Subsequent sections 
present evidence from the Cross-Site Evaluation concerning those setting, design, and implementation characteristics that are 
associated with more effective programming. 
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This monograph focuses on information that is useful to the designers and implementers of prevention programs. In part, 
this information relates to what is known about different program strategies and activities and their success in achieving spe- 
cific program objectives. This information directly relates to decisions program designers and implementers may make. These 
decisions are also shaped by the community and organizational settings in which programs operate and by the implementa- 
tion capacity of organizations and staff who deliver the program. 

One of the important objectives of the Cross-Site Evaluation is to contribute to knowledge about how prevention programs 
are acmally funded, designed, and implemented, as reflected in the experience of the programs in this smdy. In Figure 2, we 
provide an overview of major factors that operate to shape a program, and ultimately its degree of success, throughout the 
processes of design and implementation. In most prevention research, which focuses on individual programs, the close 
dynamic interaction of these factors is not considered. Evaluators assess the program as it is designed and delivered without 
the ability to consider the circumstances of setting, or of implementation, that shape the program as received by participants. 
As concepmalized here, three major sets of factors constimte these circumstances. 

Program Setting. While program strategies are a key concern of providers, programs must be designed and implemented 
within specific organizational and commumty settings. The implications of setting have often been overlooked in prevention 
research. The cross-site data allow us to investigate the influences of different community and organizational settings on the 
design, implementation, and success of smdy programs. 

Figure 2 

Factors Influencing Design and Implementation of Effective Prevention Programs 




Intervention Strategy. Before they are implemented, programs are planned. The exact content of intended programrning, 
the ways in which it will be delivered, and the amount of program contact are key elements of the intervention strategy. The 
strategy is influenced by the planning process itself, including the degree to which design is based on solid rationale or pro- 
gram theory, and the degree to which program elements are pre-specified or made explicit. 
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Program Implementation. Ultimately, program participants are affected by the program as it is delivered to them each day 
that they attend. The program that is implemented may be very much like what was planned, or it may depart from the plan 
m important ways — either intended or unintended. The capacity of the implementing organization to manage the program 
has a potentially major impact on the way in which the program is experienced by participants. 



The analysis presented below assesses the ways in which factors xvithin these three domains have influenced the effectiveness 
of programs in the National Cross-Site Evaluation. More specifically, this analysis includes the following factors. 

Ppogpam Setting 

Programs in the Cross-Site Evaluation operated in a variety of community, organizational, and service setting contexts. 

Two measures of program setting were developed — organizational setting and service setting. Each of these factors 
potentially affects program implementation and, consequently, program outcomes. 



Organizational setting refers to distinguishing features of the grantee’s organization. Profiling the characteristics of 
grantees revealed two distinct types: 



O Affiliated programs are part of a larger, resource- rich institution, such as a university, school system, or medical 
institution; and 

O Freestanding programs are community-based programs with less access to resources beyond those provided through 
the program. 

Service setting characterizes where the program’s services are actually delivered to participants. Again, two general types 

of service settings emerged from analysis of the program data: 

O In-school programs are offered during school and on school property; and 

O After-school programs are conducted outside of school hours, such as late afternoons, ev enin gs^ and/or on the week- 
ends. Some after-school programs were conducted on school property but were not part of that school’s educational 
services. Such programs were offered by grantees who obtain permission to use school property. 



Intepvenlion Sipategy 



The cross-site study includes a broad array of programs with varying content and methods of delivery. Early in the study, 
the interventions of each program were classified according to six content areas, including: 1) information (ATD [alcohol, 
tobacco and drugsJ-Direct), 2) information (ATD-Indirect), 3) academic and vocational support,. 4) targeted-skills develop- 
ment, 5) positive recreation and enrichment, and 6) emotional and social support. In addition, the delivery method of each 
intervention strategy was coded as one of four delivery methods, including 1) one-on-one, 2) didactic/classroom-style pres- 
entation, 3) support group interaction, and 4) experiential programming. These strategy and delivery categories were original- 
ly developed^ to reflect major categories of interventions that were proposed by the grantees in their grant applications. 

Over the course of the cross-site study these categories provided useful ways of describing and profiling the delivery of 
prevention services in the programs, understanding changes between program-as-planned and program-as-implemented, 
and comprehending the relationship between strategies and delivery method. 



For purposes of identifying the characteristics of effective prevention program design, however, these categories required 
simplification and focus. Drawing on the small but growing literature on generalizable guidelines to effective programming, 

the foUowing measures of important program design were developed through secondary analysis of program-level and 
dosage data: 




5 Documentaaon on the co^g procedures used for the intervendon and deUvery categories is provided in the CSAP National Cross-Site Evaluation of High-Risk Youth 
Programs Fmal Techmcal Report, available through the Center for Substance Abuse Prevention. ® 
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Program content measures were constructed to describe what is delivered to youth, that is, the major program messages. 
The foUowing four measures differentiate between those programs that emphasi2e certain content areas and those programs 

that do not;^ 



O Life skills focused programs give emphasis to developing behavioral and social skills thought to protect against substance 
use, including refusal skills, anger management, conflict resolution, decisionmaking, social skills, and academic ennch- 
ment interventions; 

O Knowledge focused programs emphasize learning about alcohol, drugs, and related topics including teen pregnancy, 
gangs, and HIV/AIDS; 

O Affective perception focused programs stress topics concerning the way youth view themselves and others, including 
emotional awareness, self-esteem bxiilding, and changing normative perceptions; and 

O Recreation focused programs devote substantial time to substance-free leisure and enrichment activities (e.g., field trips). 
Activities categorized as recreational were quite varied, including arts and crafts activities as well as sports and physical 
recreation. 



Program delivery method measures address the ways in which programs are delivered to youth. The study programs 
ranged in delivery method from primarily classroom-style, didactic programming in which youth received mformation 
through lectures, videos, or other similar means, to interactive and experiential programming in which youth were required to 
actively participate, reflect on the subject at hand, and act upon it. TypicaUy, experiential programming includes role-play 
activities, team-building projects, outdoor wilderness experiences, or other activities that require youth to be actively engaged. 
While the desirability of experiential programming as a prevention tool has been consistently supported in recent prevention 
research, the understanding of exactly what constitutes effective experiential programming, or why it is more effective, has 
not been clearly developed. The richness of the information on the National Cross-Site Evaluation program interventions 
aUows a more in-depth exploration of the nature of experiential programming and why it is effective. To elaborate on the 
delivery methods used by the High-Risk Youth Programs, three dimensions of experiential delivery methods were identified. 



O Active participation methods encourage youth to play an active role in the intervention services as opposed to passive 
involvement. Active participation includes a broad range of activities including group discussion, group projects, experi- 
ential learning activities, enrichment activities such as field trips or crafts, and recreational activities that were an integral 
part of the prevention strategy. As a category, it is distinguished primarily by its contrast to more passive forms of par- 
ticipation activities or limited participation, such as questions or adult-led discussion. Active participation is the most 
inclusive form of interactive programming identified in the study. 



O Introspective learning methods encourage youth to use self-reflection in examining their behaviors and how they impact 
others or themselves. Activities that are introspective encourage participants to think about their own beliefs and behav- 
iors and how they relate to external situations. Activities that are not introspective report information or present “desir- 
able” beliefs, attitudes, or behaviors often with the simple implication that this is the way youth should believe or 
behave. Focused role-plays or “psycho-drama” sessions are one example of introspective programming in high-risk 
youth programs. These activities typically put participants in hypothetical social situations, such as pressure to use, a vio- 
lent confrontation, or an embarrassing situation. Participants act out scenarios, sometimes on tape, and then discuss their 
actions, feelings, and outcomes within the group. Another example would be peer or adult-facilitated group discussions 
in which participants reflect on real-life actions they have taken. Examples include activities designed to help youth 
understand their value and impacts on others (e.g., older adults in cross-generational programs), chaUenging activities that 
encourage youth to reflect on their abilities and contributions (e.g., ropes courses and wilderness training), or program 
themes that focus on understanding one’s relationship to significant others and society. 



6 Programs were placed in these four measurement categories through the review of data on program interventions and data on the relative amount of contact youth had 
with each of the intervention types. The measures described the dominant category of programming offered to youth. Life-skills dommant programs are those that 
focused primarily on targeted sldll development, academic and vocational support, or positive recreation and enrichment that focused on skill-development (e.g., wilder- 
ness adventures, ropes courses). The exception to this coding decision was in the area of affective self-esteem buUding programming, which became a new strategy type, 
in combination with programs that focused on emotional/social support. Knowledge-dominant programming combines the two informational strategy types. 

0 n 
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o Connection building methods involve techniques such as team building to help youth connect with others versus individ- 
ualistic learning approaches. The distinguishing characteristic of connection building methods is that they involve partici- 
pants in situations in which their interactions with others are a central point of accomplishing the activity. Examples of 
these connection building activities include projects or service activities from which the youth have pla nning and imple- 
mentation responsibility (e.g, special events, community projects, performances), activities focusing on group or dyad 
interactions (e.g, parent-child programs, mentoring relations), or program themes that focus on mutual support and con- 
nection. 



Program Implementation 

The effectiveness of an intervention depends on how well it is put into action. Evaluations of prevention and other social 
programs have clearly demonstrated that many programs did not meet their expectations because they were not delivered as 
planned. The Cross-Site Evaluation includes information on three dimensions of program implementation. 

Service contact measures assess the amount of services provided to youth. Service contact, or dosage, is a fundamental 
implementation dimension. Two measures of service contact were developed, including: 

O Amount of contact is the average number of days that youth received prevention services at each program, and 
O Intensity of contact is the average number of hours of services per week received by youth at each program. 



Program structure measures describe the degree to which programs are clearly and consistently structured. The following 
two variables were created to assess program structure: 

O Program coherence is the degree to which program implementation was guided by a clear theoretical framework, and 
O Program consistency is the degree to which programs were implemented regularly and with consistency. 



Program management measures assess the extent to which management is able to create a supportive and empowering 
work environment for staff. Two measures of management were developed. 

O Supportive management is the degree to which the program produces a work environment where staff feel supported 
and satisfied with their work, and 

O Results-oriented management is the degree to which staff perceive the program is managed in such a way that they are 
effective in achieving positive outcomes for the participating youth. 

For prevention practitioners and policymakers, the importance of prevention evaluation lies primarily in the direction it 
can provide for future decisions. Furthermore, this direction must be applicable in the diversity of real settings in which 
programs are implemented. This monograph uses the rich program information generated by the Cross-Site Evaluation 
to develop measures relevant to practice. These measures will be used in the next section, which reports findings on the 
effectiveness of different program design and implementation strategies in reducing substance use and strengthening 
protective factors among youth at high risk. 
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The central question for this monograph is to determine whether the program characteristics identified above distinguish 
between more and less effective programs. The findings reported here are from analyses that used data collected when par- 
ticipants entered their program and when they completed the program. These findings are sometimes referred to as immedi- 
ate program outcomes. The primary outcome measure used in these analyses is an overall measure of substance use, which 
combines 30-day cigarette, alcohol, and marijuana use. When they are important to help elaborate on the findings, five more 
specific outcomes are used: 1) 30-day cigarette use; 2) 30-day alcohol use; 3) 30-day marijuana use; 4) school connectedness; 
and 5) family connectedness. Substance use attitudes are not included in these analyses because prior analysis demonstrated 
that program effects on substance use attitudes are not related to program effects on other outcomes. In other words, the 
research showed that increasing negative attitudes toward substance use does not lead to reduced use of alcohol, tobacco, 
and/ or illicit drugs. 

Effects of Progpem Setting Factors 

Programs operate within particular organi 2 ational environments. While these environments clearly create opportunities 
and constraints that shape the day-to-day opportunities of staff, there is little systematic information on the ways in which a 
program’s setting impacts its design and implementation. Indeed, few studies encompass differences in setting sufficient to 
support such an analysis. The Cross-Site Evaluation offers an exceptional opportunity because it includes programs in a 
variety of organizations that are delivered in different settings. 

Organizational Setting. The organizational setting of programs may shape the capacity to implement prevention services. 
The degree of affiliation that programs have with large institutions potentially impacts their resources and may affect the 
ways in which prevention programs are designed. Programs that have greater access to resources, including additional facili- 
ties and staff, may be able to more successfuUy deUver services. However, freestanding, nonaffiliated programs that are locat- 
ed within the neighborhood of the target population and that have greater familiarity with the issues facing that population 
may benefit from greater experience with and proximity to the target population. 

In the cross-site study, about two-thirds (62%) of participating programs were classified as freestanding, and a little more 
than one-third (38%) were categorized as affiliated programs. In Figure 3, the differences in resource availability between 
affiliated and freestanding programs are profiled. Across aU resource areas, affiliated programs have greater access to 
resources than freestanding programs. 
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Figure 3 

Percentage of Affiliated and Freestanding Programs Receiving Categories of Resource Assistance from Grantee Organizations 

(N = 48) 




Figure 4 displays the differences in effect si 2 es between affiliated and freestanding programs^ The higher the effect si 2 e, the 
greater the positive impact the programs had on reducing substance use among participating youth. As shown in the graph, 
there is only a small, statistically insignificant difference in 30-day substanceS use in the effect si 2 es between programs a dmin - 
istered by affiliated and freestanding organkations. Although affiliated programs have more organkational resources at their 

command, fireestanding programs in the community are able to design and implement programs with similar effectiveness for 
preventing substance use. 



Figure 4 

Average Effects on 30-Day Substance Use At Affiliated and Freestanding Programs 

(N = 46) 





’ Median effect sizes for this and subsequent figures are presented because effect sizes ate not notmaUy distributed. 

The Wilcoxon test (one-tailed) is used to determine statistical significance in aU subsequent figures presented in this report 
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Service Setting. The optimal setting for deUvering prevention services to youth has been a matter of debate among 
researchers and practitioners. School-based services have clear advantages of access, retention, and potential scale. However, 
their effectiveness for youth at higher levels of risk has been questioned because school-based services are usually umversal 
approaches geared to the general student population. In the National Cross-Site Evaluation, 1 5 programs dehvered a sub- 
stantial portion of their prevention messages in school classrooms, and 33 programs dehvered services after school. This 
diversity in setting allows a comparison of effectiveness on substance use and select protective factors. Figure 5 compares 
average effectiveness on 30-day substance use’ achieved by programs that were dehvered in school and those dehvered after 
school. Differences for 30-day use approached statistical significance, with after-school programs producing more positive 
effects than in-school programs. 



Figure 5 

Average Effects on 30-Day Substance Use at After-School and In-School Programs 

(N = 46) 




Programming 30- day USG 

y After-school (n=33) 
I In-school (n= 13) 



Figure 6 shows effect of service setting for the individual substance use outcomes. The difference between effect si 2 es for 
in-school and after-school programs is significant for cigarette use, with after-school programs producing a much more 
positive effect on cigarette use. Also, the pattern of effects suggests that after-school programs were more effective for 
alcohol use as well. 




Figure 6 

Average Effects on Individual Substances at After-School and In- School Programs 

(N = 46) 




I In-school (n>=13) 



* Statistically significant at 0.05 level (one-tailed Wilcoxon test). 



^ When using substance use outcomes in the analysis, 
prevented access to alcohol and other drugs. 



the number of programs is reduced to 46 because two programs were implemented in controlled environments that 
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A contrast in the effectiveness of in-school and after-school programs with respect to school connectedness and family con- 
nectedness is shown in Figure 7. While after-school programs have positive effects on family connectedness, the in-school 
programs show positive effects on school connectedness. 



Figure 7 

Average Effects for Risk and Protection Measures by Program Setting 
(N = 48) 




Bl After^chool (n 33) B In-school (n= 15) 



* Statistically significant at 0.05 level (one-tailed Wilcoxon test). 



School connectedness is the sole measure for which in-school programs are more successful than after-school programs. 
Effects are particularly strong in programs with high intensity, high levels of academic support activity, or high levels of 
one-on-one activity. The findings strongly suggest that the classroom may not be as effective in a setting direcdy influencing 
substance use among high-risk youth. However, progranuning in school does appear to have a positive impact on youth’s 
perceptions of connectedness to school, an important protective factor. 

The differential effectiveness of in-school and after-school programs may be attributable to the particular opportunities and 
constraints that characteri 2 e each service setting because the setting can shape the type, style, and amount of service that is 
offered. Programs offered during school hours are constrained by the periods in which they are offered, while after-school 
programs have more discretion in their ability to shape program offerings. The nature of the school setting requires planning 
and scheduling to fit into the classroom routine and environment. The constraints of schedule and competing classroom 
demands may limit the amount of total service time, restrict peer bonding that might develop in more intensive after-school 
programming, and be oriented toward less experiential and more didactic delivery methods. 

Figure 8 summari 2 es differences between in-school and after-school programs on several program delivery characteristics. 
The differences are clear. In-school programs provided less contact with youth than after-school programs, had less orienta- 
tion to building relations in their activities than after-school programs, and had fewer activities requiring active participation 
(each of these characteristics is explained more fiilly in subsequent sections of this report). This pattern is consistent with 
the opportunities and constraints in each setting and may help explain differential effectiveness across the settings. 
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Figure 8 

Program Characteristics of In-School and After-School Programs 
(N = 48) 




* Statistically significant at 0.05 level (one-tailed WUcoxon test). 
** Statistically significant at 0.01 level (one-tailed WUcoxon test). 



Effects of tntervention Stpategy Factops 

When practitioners think about designing prevention programs, many of their questions concern intervention strategy or 
program content and delivery. What types of learning experiences or activities are most effective in preventing substance 
use? What are the most effective ways to deliver these services? This section explores each of these questions. 

Program Content. The participating study sites used an array of program designs with different emphases and learning 
methods, thus providing an excellent opportunity to test differences in effectiveness based on program content. As noted 
above, descriptive program information was used to identify the degree to which study sites used each of four different cate- 
gories of program content in their activities. Figure 9 displays the number of programs in which each type was emphasized 
more than others. For example, a program is categorized as knowledge-focused if participating youth were exposed to this 
prevention strategy more than any of the other three. Any or all of the others, however, may have been used in the program. 



Figure 9 

Number of Programs By Program Content 
(N = 48) 





23 

o 

ERIC n B U an 0 S is as 8 0 8 si si n 0 0 1 8 1 m 0 8 1 i li $0001000 Mslmini0!P8iB00 








ls[p0§8«@ MiWffl (SO ' mmttlii 



When categorized this way, the largest group of cross-site programs (n = 17) was knowledge focused. While these programs 
often included other types of activities, they emphasized the presentation of information about substance use and other 
related problem behaviors. These programs often emphasized negative health and social consequences associated with sub- 
stance use. 



The second largest group of programs (n = 14), emphasized life-skills training. These interventions focused on specific atti- 
tudes and behavior skills often identified as “protective” in the risk and protection literature. As applied in the study pro- 
grams, these skills include refusal skills, anger management, communications skills, decision and goal-setting skills, academic 
and vocational skills, and other social skills. The category also includes ^^recreation and enrichment” activities that focused on 
skills development (e.g, wilderness programming, ropes courses, programs of arts with competency emphasis). 

The third category (n — 12) placed priority on affective programming. These programs focused on emotive and awareness 
factors thought to be associated with substance use. Following Tobler (1993), affective programming includes activities 
focusing on self-esteem, self-awareness, and attitudes, beliefs, and values associated with substance use and other problem 
behaviors. 



Finally, a small number of programs (n = 5) emphasized substance-free recreational progranuning that focused on alcohol 
and drug-free leisure time activities, often including organized sports. 



Effects of Program Content on Substance Use. A major purpose of the National Cross-Site Evaluation was to test 
whether the nature of prevention programming makes a demonstrable difference in the effectiveness of progr am s for 
reducing substance use among high-risk participants as compared to non-participating comparison youth. Figure 10 
clearly indicates that the primary content strategies of programs did make a difference. Programs that focused on 
delivering life-skills programming were significantly more effective than other types of programs, with recreation 
programs the second most effective. Programs that focused on providing knowledge about alcohol, drugs, and other 
attendant issues were significantly less effective than the other types of programs. These fin din gs support prior research 
on prevention strategies for youth ^obler et al., 2000) that show the superiority of life-skills content over programming 
that focuses on other information only, or on affective orientations. Importantly, the National Cross-Site Evaluation has 
confirmed these findings for a large number of programs targeting high-risk youth. 



Figure 10 

Average Effects on 30-Day Use by Program Content* 
(N = 46) 
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* Statistically significant at 0.05 level (one-tailed Wilcoxon test). 
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Effects of Program Content on Risk and Protective Factors. Figure 11 displays differences between the four program 
types in their effectiveness for strengthening school connectedness. Even though these differences are not significant, the 
pattern across program content types is the same as that for substance use. The patterns suggest that life-skills programming 
is a particularly effective strategy. However, the differences between the other three program types are small. 



Figure 11 

Average Effects on School Connectedness by Strategy Type 
(N = 48) 




Delivery Method. Information about the Cross-Site Evaluation programs shows that the same program content may be 
delivered in different ways. Messages about how to resolve conflicts, for example, may be delivered through lectures in which 
the participant is largely a passive recipient or in more experiential methods such as role plays. Past studies have pointed to 
the method of service delivery as an important factor in the effectiveness of prevention programming. In particular, deKvery 
methods that involve experiential processes, particularly among the youth themselves, have been shown to be more effective 
than didactic teaching or adult-led techniques (Hansen, 1992; Tobler, 1986; Tobler and Stratton, 1990; Tobler et al., 2000). 

As explained above, the Cross -Site Evaluation data allow delivery methods to be differentiated in several ways including 
(1) the degree of interactive participation required; (2) the degree of introspection (self-examination) required; and (3) the 
amount of emphasis placed on connection building. These three concepts help to elaborate experiential programming and 
to understand why it is an important component of effective prevention programs. 

1. Interactive Participation. As introduced earlier, the first type of experiential programming simply distinguishes the degree 
to which the prevention activities in a program require active or passive participation by program youth. As shown in 
Figure 12, the majority of the programs (n = 31) were categorized in the higher, more interactive participation group. 
They did not rely on passive learning techniques to convey their prevention message. 

Figure 12 

Distribution of Experiential Delivery Methods 
(N =48) 
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2. Introspective Participation. A second measure of experiential participation identifies 18 of the active programs as introspec- 
tive in their programming. This dimension characterizes participation in prevention activities as the degree of self-aware- 
ness or reflection that is facilitated through those activities. Eighteen (38%) of the Cross-Site Evaluation programs were 
classified as having a more introspective orientation. 

3. Connection-Pui/ding Participation. Another subcategory of interactive participation distinguishes between programs that seek 
to build a positive social context and interaction among participants and that seek to increase the awareness and skills of 
youth through collective activities such as discussion groups, group exercises, and team projects. This category also 
includes programming specifically focused on building supportive relations with an adult (e.g, mentoring activities). 
Around one-quarter (n = 13) of the cross-site programs were categorized as connection-building. 

Effects of Delivery Method on Substance Use 

Each of the dimensions articulated above represents an elaboration of the ways in which youth may interact with peers and 
leaders within their programs. Figure 13 presents median effect sizes for 30-day substance use within programs that are high- 
er and lower on each of these dimensions. 



Figure 13 

Average Effects on 30-day Use for Delivery Method Measures 
(N = 46) 




* Statistically significant at 0.05 level (one-tailed Wilcoxon test). 
** Statistically significant at 0.01 level (one-tailed Wilcoxon test). 



Programs that emphasize connection building had significantly higher effect sizes than programs that focused on 
individually oriented strategies and activities. The extent to which programs helped youth understand their own 
orientations and behaviors was also important, with significantly higher effects for programs that were oriented toward 
introspective activities. Active participation was less influential, although patterns of effect sizes for youth in programs 
with active rather than passive activities suggest that these programs were more effective than programs with a more 
passive orientation. In sum, programs that actively engage youth in thought-provoking and meaningful activities that 
encourage building positive connections to peers or adults are most likely to produce positive effects. 
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The cross-site analysis also explored the degree to which the elements of positive program design reinforce each other. For 
this analysis, a combined measure of all three types of interactive programming was used. When both program content and 
method of delivery are considered, programs with both a life-skills orientation and interactive methods are significantly more 
effective than other programs. The median effect si 2 es for programs characterized by both life-skills programming and inter- 
active delivery are higher than those for programs that combine non-interactive methods with life-skills development (see 
Figure 14). In short, programs that engage at-risk youth experientiaUy in skills development activities have the greatest impact 
on reducing substance use. 



Figure 14 

Average Effects on 30 -Day Substance Use by Interactive and Life -Skills Focus 

(N = 46) 



More Effective 
Programming 



Less Effective 
Programming 




r~1 Life Skills More Dominart (np12) 
^ Life Skills Less Dominant (np34) 



* Statistically significant at 0.05 level (one-tailed Wilcoxon test). 



Effects of Delivery Method on Risk and Protection 



As shown earlier, life-skills development strategies also have a positive effect on school connectedness, which is an important 
protective factor. When service delivery methods are factored into the analysis we find similar patterns as those shown for 
30-day substance use. Figure 15 shows highly significant differences in effect sizes for school connectedness depending on 
whether programs emphasized introspection, connection building, and active learning. Youth who are asked to participate in 
programming in meaningful ways clearly improve their own school connectedness. When building connectedness with other 
youth in prevention programs is a program focus, connectedness to school also improves. 



Figure 15 

Average Effects on School Connectedness for Delivery Method Measures 
(N = 48) 



More Effective 
Programming 



Less Effective 



Programming 




** Statistically significant at 0.01 level (one-tailed Wilcoxon test). 
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In summary, the Cross-Site Evaluation confirms and extends the accumulating evidence on effective design of prevention 

programs. The findings specify the types of interventions and delivery methods effective in working with high-risk youth. 

O First, the strong patterns of difference by program content confirm that knowledge-focused and affective programming 
do not independendy reduce substance use among high-risk youth. These findings are consistent with findings from 
studies on the general youth population. 

O The findings indicate that life-skills programs that consider a variety of academic, social, and personal skills are effective 
for youth who are at high risk for substance use. Well-designed, substance-free recreation programs are also promising 
for high-risk youth. 

O Method of delivery has also been shown to contribute to program effectiveness. The findings confirm that non-interac- 
tive delivery methods are not effective, and help elaborate the nature of the interactive methods that are effective. In par- 
ticxilar, the findings suggest that at-risk youth benefit from programming that helps them reflect on their own behavior 
in social contexts and to build relationship skills through group or individual interaction. 



Effects of Program tmplementatioo Factors 

The evaluation field has clearly demonstrated that there is often considerable slippage between program planning and the 
program services that are actually received by participants. The process of implementing program ideas will inevitably deter- 
mine the exact configuration of the program experienced by youth. It follows that improved understanding of this complex 
topic is important to improving prevention effectiveness. 

This section includes information on three areas of program implementation. First, the importance of the amount and 
intensity of program contact for prevention effectiveness is assessed. Second, the importance of clear and consistent organi- 
zational and conceptual structure of activities for effective programming is considered. Finally, indicators of successful man- 
agement are related to program effectiveness. 

Service Contact. If program activities are to make a difference in the lives of youth, they must be organized and delivered 
so that youth participate sufficiendy to benefit. Programs that do not provide a sufficient dose of service presumably cannot 
impact youth. For the following analyses, three measures of the contact that youth had with their program are used. First, we 
consider the total amount of contact (number of hours), next, the length of time in the program, and last, the intensity of 
contact, or the number of hours per week. 



O Effects of Amount and Length of Time in Program. The cross-site study collected individual program contact data on each 
participating youth. For each of the program interventions, the amount of service received was recorded for every day 
that the youth participated. Data were then aggregated to compare the amount of service provided, on average, for each 
of the cross-site programs. Analyses of average amount of program contact showed no significant relationships with 
effect size for any of the outcome measures of interest. 



The Cross-Site Evaluation also provided data on the length of time in weeks that youth participated in prevention pro- 
gramming. Analyses of the length of time youth participated in program services showed no significant relationships 
with any of the outcome measures. These findings suggest that average amount of contact and length of contact with- 
out considering their nature and quality are not powerful contributors to prevention effectiveness. 

^ ^ff^^i^ ff Program Intensity. Program intensity is measured as the number of hours of contact per week of programming. 

Across the cross-site programs, intensity ranged from programs that averaged less than 1 hour of service per week, to 
programs offering 15 hours of service. Figures 16 and 17 display the differences in effect sizes at program sites with 
high and low amounts of intensity by the major outcome variables. Programs were divided into two equal groups — 
higher intensity (3.3 hours per week or more) and lower intensity (less than 3.3 hours per week). 
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Figure 16 

Average Effects on Substance Use Measures by Program Intensity 
(N = 46) 



0.12 




Less Effective Cigarettes Marijuana 

Programming 

g Higher Irtensity (n=23) | Lower htensity (n= 23) 



* Statistically significant at 0.05 level (one-tailed Wilcoxon test). 



Figure 17 

Average Effects on School and Family Connectedness by Program Intensity 
(N = 48) 



More Effective 
Programming 




Less Effective 
Programming 



0.04 




-0.08 1 1 

I Family Connectedness 

School Connectedness 

Bi Higher Intensity (n°24) H Lower Intensity (n°24) 



Program intensity, unlike total amount of program contact, has a significant relationship with major outcome variables. 
More intense programs were significantly more effective in changing alcohol and 30-day substance use patterns. There were 
no significant differences for the other outcome measures though the pattern suggests the importance of program intensity 
for school connectedness. 

In summary, the intensity of programs, or the number of hours per week of exposure they provide, has a sigmficant 
relationship to substance use of at-risk youth, but total amount of contact does not. This finding suggests that program 
impact is facilitated by a concentrated presence in the lives of youth. 
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Program Structure. Prevention programs vary in their degree of explicit rationale and organization to accomplish defined 
outcomes. Indeed, one of the objectives of CSAP grant funding procedures has been to increase this explicit program struc- 
turing through the use of logic models and similar tools. The Cross-Site Evaluation analysis considers the impact of two 
types of structuring on program effectiveness-coherence and consistency. Coherence relates to conceptual organization and 
rationale, and consistency relates to the clear structuring of program time and schedule. 

O Effects of Program Coherence. As noted by Hansen (1997), most prevention programs are fashioned by practitioners who 
construct “bundles'" of programming that are pulled into programs for a variety of reasons based on preference, avail- 
ability, or belief as well as science-based merit. The cross-site measure of program coherence refers to the extent to 
which program theory is explicit, articulated, and used to focus multiple activities on program objectives. Some programs 
have a concrete theoretical framework and guiding principles that are understood and followed by all service deliverers. 
Other programs are more loosely structured where staff work independently from one another with no central guiding 
theory or method. Analysis of the site visit data allowed us to categorize the programs as exhibiting higher (n = 18) or 
lower (n = 28) coherence in their program rationale and activities. 

Programs structured by a clear purpose and strategy were more effective in changing substance use patterns, particularly 
for alcohol use, than less coherent programs. Figures 18 and 19 show the distributions of effect sizes for substance use 
and risk and protection measures based on program coherence. 



Figure 18 

Average Effects on Substance Use Measures for Programs with High and Low Coherence 

(N = 46) 



More Effective 
Programming 




Less Effective 
Programming 




H High Coheren ce (n ° 18) 
I Low Coherence (n ° 28) 



* Statistically significant at 0.05 level (one-tailed Wilcoxon test). 
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Figure 19 

Average Effects on School and Family Connectedness Measures for Programs with High and Low Coherence 

(N = 46) 
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While differences for school connectedness or family connectedness are not statistically significant, the patterns suggest that 
these protective factors are influenced more positively in programs with higher coherence. This finding reinforces the impor- 
tance of clear program purpose and logical connection of program activities to that purpose. Funding and technical assis- 
tance for prevention programs should encourage consistency, planning, and training that supports this program quality. 



O Effects of Program Consistency. Consistency in program structure varied widely across the cross-site programs. A number of 
the programs met after school on a regular basis, and were organi 2 ed with consistent activities that included skill devel- 
opment, homework assistance or tutoring, and/ or recreational activities. In-school programs often included weekly or 
biweekly programming during class hours. Other programs, however, were more loosely structured. For example, they 
might have been organi 2 ed around services with a loose case management format, where youth would meet with staff 
on a periodic basis, but in an unstructured format with less consistency than programs with regularly scheduled services. 
The analysis showed that there were no statistically significant differences between the more and less consistent pro- 
grams for any of the outcome variables. 



O Effects of Program Consisteng/ and Coherence. Although program consistency by itself does not contribute to program suc- 
cess, an interesting pattern emerges when consistency and coherence are combined. As shown in Figure 20, programs 
that are both more coherent and more consistent are more likely to be effective than other programs, particularly com- 
pared to programs with both minimal coherence and consistency. 




Figure 20 

Average Effects on 30-Day Use Measures by Consistency and Coherence 
(N = 46) 



More Effective 
Programming 



Less Effective 
Programming 



0.20 




B High consistency, high coherence (h = 10) 
I Lew consistency, high coherence (n = 8) 

I I High consistency, low coherence (n = 9) 
I Low consistency, towcoherence (n=19) 
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In summary, the pattern of program effects displayed in Figure 20 has two implications. First, coherence of program theory 
and activities is not the same as consistency in organization and schedule. Programs high on one dimension might be low in 
the other. Second, each quality contributes to effectiveness. When both are present, programs are most effective. 



Program Management. Organizations have their own ways of operating that shape program delivery. All organizations 
develop “routines, or standard operating procedures, to guide workers’ actions in different circumstances” (Scheirer, 1987). 
These routines may be guided by written rules or program guidelines, through strong leadership from supervisors or through 
the norms and standards about how work is accomplished, that are brought in by staff or are developed in the program 
work group. 

These aspects of the program environment are undoubtedly related to management. The clarity of job functions and 
responsibilities, the degree to which procedures are explicit and rule oriented, the nature and clarity of leadership, personnel 
policy, and the opportunities for staff to have input and shape the program are all examples of organizational operations 
that are strongly shaped by management. The experience of CSAP-funded demonstration programs, and that of other fun- 
ders of demonstration programs, has amply shown that programs often fail because of organizational failure, breakdowns in 
the personnel systems (e.g., high turnover, burnout) or organizational operations that prevent implementation of the services 
proposed by the program. However, management procedure that is effective in one organizational setting or work group 
may not be effective in another. The cross-site program sample encompasses heterogeneous organizational settings and 
management styles that are likely to differ across program settings. 

Considering the clear importance of management, the following analysis focuses on two components of organizational oper- 
ations that are expected to contribute to effective program implementation-positive and supportive work environment and 
results-oriented management. Both of these components assess effective program management by the degree to which they 
empower staff to feel supported and effective in carrying out their responsibilities, rather than by a direct assessment of how 
these management results were achieved. 

^ Positive and Supportive Work Environment. The first management component indicates the degree to which a pro- 

gram has been designed and managed to produce a work environment in which staff feel supported and satisfied with 
their work. This management variable has a complex relationship to program effects. Although the analysis found no 
simple linear relationship between a positive work environment and the degree of program effectiveness, this manage- 
ment component does seem to define a threshold of program capacity for effectiveness. While providing a positive work 
environment does not guarantee program success, none of the programs in the cross-site study that had low work envi- 
ronment scores produced positive effects, particularly for risk and protective factors. On family connectedness, for 
example, all programs showing positive effects of 0.2 or greater score above 75 on this management variable (on a scale 
of 1 to 100). This finding suggests that providing a positive work environment is a necessary but not sufficient condition 
for an effective program. WeU-nianaged programs also need other effective strategy and implementation characteristics 
to impact substance use among at-risk youth. 

O Effects of Results-Oriented Management. The second management component indicates the degree to which staff perceive 
that the program has been designed and managed to allow them to feel that they are effective in achieving positive out- 
comes for participating youth. The analysis showed that in programs where staff felt that they were having positive 
results, school connectedness was significantly more improved than in programs where staff felt less empowered. Similar 
patterns were present for alcohol use, marijuana use, and family connectedness, although these differences were not sta- 
tistically significant (see Figures 21 and 22). 




32 




IS s alB p 



PPBuenUois 



Points Ot !FD“®TO1|®[I ' ■ ■ 

St’rv^ I.IUnP^^^n mpn^ I c:^^nrr,^nfl^r-, _n Mlflrnl^ fnflr^^h W^^mn^^ lllm^fnmorr,o 





Jlaii0iii0D iSi?0§§“8lfl0 so LIJD 


pliUMi 


[}'P0P0i§ 



Figure 21 

Average Effect Size on Result-Oriented Management by Substance Use Measures 

(N = 46) 



0.05 




Less Effective -0.04 -* i | i | 

Programming | Alcohol I 30-day Use 

Cigarettes Marijuana 



I Strong results-oriented management (n=23) 
I Weak results-oriented management (n=23) 



Figure 22 

Average Effect Sizes on Risk and Protection Measures by Results-Oriented Management 

(N = 46) 



More Effective 
Programming 




Less Effective 
Programming 




School Connectedness* Family Connectedness 



I Strong results-oriented management (n=23) 
I Weak results-oriented management (n=23) 



* Statistically significant at 0.05 level (one-tailed Wilcoxon test). 



These findings suggest that both supportive and results-oriented management are components in program excellence, but 
they cannot substitute for other basic building blocks of intervention strategy and implementation. 
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Summary of Findings 

An underlying purpose of the National Cross-Site Evaluation of High-Risk Youth Programs is to produce and disseminate 
practical lessons on how to better contribute to the positive development of youth at high risk. This monograph represents 
the first step in reali 2 ing these objectives. Most generally, the monograph is encouraging in its promise that science-based 
principles of good programming can be identified across a broad range of programs serving high-risk youth. The findings 
are consistent with findings from other studies in some areas. The importance of active participation by youth and the com- 
parative efficacy of life-skills program content are examples. The Cross-Site Evaluation has added new lessons in other areas. 
For example, the study provides new understanding of what makes interactive participation effective by identifying the com- 
ponents of that idea. The study also indicates that it is not exclusively program strategy that matters; other important factors 
also contribute to effective programming. More specifically. Figure 23 summarizes program characteristics that contribute to 
significant reductions in 30-day substance use. 



Figure 23 

Summary of Significant Program Effects for 30-Day Use 
(n = 46) 



a 18 




Illustrated in the above graph are the following points: 



O Program Content. Program content is critical to improving adolescent behaviors. Programs with strong life-skills program- 
ming were clearly more effective than programs emphasizing other content in changing substance use and school con- 
nectedness. Recreation-focused programs also demonstrated patterns of positive effect, though the number of programs 
was small. Programs that focused on providing information about cigarette, alcohol, and other drugs were significantly 
less effective. The weakness of knowledge-only and affective programming has been recognized in prevention research, 
and IS confirmed again in this study of programs serving youth at high risk. Future program development should focus 
on life skills and positive alternative approaches. 



O Program Delivery Method. These findings confirm and elaborate the importance of the way in which prevention messages 
and lessons are conveyed in programs. Clearly, as demonstrated in other research and reiterated here, effective program- 
ming must use interactive, rather than passive, classroom-style learning methods. Furthermore, techniques that focus on 
building positive connectedness with peers or supportive adults, and methods that encourage youth to think through 
their own positions and circumstances are promising. Effective interactive activities should be a focus of future program 
development. 
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O Program Coherence. Programs with coherent program theory that includes clear links between outcome objectives and pro- 
gram activities are more effective than programs with a less clearly articulated rationale. The positive outcomes of coher- 
ent programming are enhanced by consistent well-organized schedules of activity. These findings suggest that CSAP’s 
emphasis on science-based programming with clear, evidence-based rationale for program activities has improved pro- 
gram effectiveness and should be a continuing focus in planning and implementing prevention programs. 

O Contact with Youth. The average amount of program contact was not related to program effectiveness, suggesting that the 
amount of contact is less important than other attributes of effective programs identified here. However, programs with 
more intense contact (i.e., more hours per week) achieved more positive outcomes. Program planners and implementors 
should ensure sufficient intensity of contact to have a meaningful presence in participants’ lives. 

O Service Setting. After-school programs were found more effective in changing cigarette and alcohol use and family con- 
nectedness than programs offered primarily during school hours. Findings suggest that this difference can be understood 
primarily as a function of the constraints in the school setting that limit program intensity and interactive programming. 



The pattern of findings has important implications for strengthening prevention programming for high-risk youth as imple- 
mented in communities across the Nation. 
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For prevention program designers and implementers, the ultimate test of the utility of science-based knowledge lies in the 
ability to go beyond identifying what to focus on as program objectives, and to help identify exactly how to work with youth 
to achieve those objectives. Through collecting detailed data, and relating program characteristics to outcomes achieved by 
the study programs, the National Cross-Site Evaluation has produced science-based answers to practitioner questions about 
how to make prevention activities more effective. 

Ppogpam Content 

Cross-Site Evaluation programs that focused on developing life skills were more effective in reducing substance use than 
programs that emphasized other program content. Programs that focused on knowledge and information concerning alco- 
hol, drugs, and other related issues were significandy less effective than other programs, as were programs that emphasized 
affective programming concerned with changing attitudes, self-esteem, and value perceptions. These findings are consistent 
with recent meta-analyses of prevention programs (Tobler et al., 2000). 

O Prevention programs need to be designed to support each youth’s relationships with positive social environments. 

Generally, prevention strategies should promote opportunities and skills that build connection between youth and posi- 
tive environments. While family and school are clearly important examples, caring adults other than parents, positive 
peers, faith communities, and other positive contexts serve similar functions. Program activities should be used to help 
youth connect to these social contexts, and understand and appreciate their contributions in these contexts. 



Ppogpam Delivepy Methods 

Cross-Site Evaluation programs that involved participants interactively were more effective in reducing substance use than 
programs that relied more on passive, classroom-style learning, confirming similar findings in meta- analytic studies (Tobler 
and Stratton, 1997; Tobler et al., 2000). The cross-site analysis elaborated on this finding by identifying component styles of 
interactive programming and assessing their relative contribution to program effectiveness. Among the different types of 
interactive methods, connectedness -building approaches that focus on activities in which youth work cooperatively with 
adults or in peer teams made the strongest contributions. This finding is consistent with the connectedness theme running 
throughout the cross-site findings. Approaches that helped youth understand their own orientations and behaviors were also 
more effective than methods that simply involved active participation. 

O Programs should be designed to actively engage youth in thought-provoking and meaningful activities that encourage 
team building. Program planners should avoid didactic activities in which youth “receive” information through 
lectures, videos, or other similar means. Classroom-style approaches where youth work individually on activities 
should be de-emp hasized. 

Program Contact with Yonth 

The Cross-Site Evaluation team collected detailed information on the duration and amount of participant contact with their 
program. Analysis of this information demonstrated that programs with higher levels of contact were on average no more 
successful than programs with less contact. However, more intense programs, measured as programs delivering more hours 
of programming per week, were more effective in reducing substance use than programs that were less intense. The total 
number of hours per week spent by youth in a program is more important than the overall amount of time youth are 
exposed to program services. 

O Program planners should design programs that provide 4 or more hours of service per week. This intensity of service is 
apparendy more important for outcomes than duration or total number of hours of contact. 
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In-School Progroms 

The study findings support the conclusion that the classroom is not the best setting in which to provide a substance abuse 
prevention curriculum. However, the findings also demonstrate the importance of school connectedness as a protective fac- 
tor, and the greater effectiveness of in-school programs in achieving stronger school connectedness. The latter finding is 
consistent with the findings of the Child Development Project, a CSAP model program, that demonstrates the many positive 
benefits of changes in the school environment to make it more of a “caring commumty” (Battistich et al., 1996) in the eyes 
of students. 

O To enhance the protective influence of schools and to increase the numbers of young people who are supported 

through a protective environment, programs offered in schools should focus on building positive relationships among 
youth and adults and providing youth with a caring community in which they can sustain meaningful relationships. 



Program Management 

Programs must be effectively implemented if they are to deliver services as planned. In the Cross-Site Evaluation programs, 
positive management practices appeared to provide a necessary but not sufficient foundation for program effectiveness. 
Supportive management, measured as management practices that provided staff with resources, training, voice, and work 
conditions perceived as empowering them to be effective in their job, did not guarantee program success, but programs with 
poor management rarely produced positive effects, particularly for the risk and protective factors measured in the study. 
Above this floor, programs that were coherent in relating clear understandings of objectives, planned activities, and the 
relationship between these two factors to staff, were more effective in achieving outcomes. In sum, programs with coherent 
planning and management were more effective. 

O Program administrators and managers need to plan coherent programs with clearly articulated objectives, and sound rea- 
soning for selecting activities and strategies to achieve those objectives. Managers must work with their staff to under- 
stand program objectives and rationale and help them feel that they are having positive results with the youth they serve. 



Program Outcome Objectives 

The degree to which youth express more or less accepting attitudes toward substance use has often been identified as a 
risk and protection factor, and moving attitudes toward less acceptance has often been identified as a program objective. 
Achieving this desired change in attitudes has been considered a sign of program success. However, the Cross-Site 
Evaluation findings clearly demonstrate that programs that achieve positive effects on attitudes (i.e., more reports of non- 
accepting attitudes) are no more likely than other programs to produce positive effects on the other primary outcome vari- 
ables assessed in the study. In contrast, positive program effects for all of the other primary outcomes tended to co-occur. 
Furthermore, the kinds of programs that were effective in changing reported attitudes were different than the kinds of pro- 
grams that were effective in changing the other risk and protection factors and substance use behaviors. The conclusion and 
the implications for program outcome objectives are fundamental. Program interventions that are more effective for chang- 
ing attitudes are not more effective in reducing substance use or strengthening important risk and protective factors meas- 
ured in this study. 




O Program managers should not consider changes in attitudes alone to be an indicator of program success, and program 
designers and implementers should not design programs to specifically achieve attitude change. Rather, programs should 
be designed to directly maximize change in the other primary outcomes identified in the Cross-Site Evaluation. 
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This monograph has presented information on program characteristics that contribute to the effectiveness of prevention 
programs targeting high-risk youth. The findings begin to unravel the complex knot of interconnected influences of setting, 
organkation, and program design that can strengthen or diminish program effects. They also begin to provide important 
concrete guidance concerning what elements of design or implementation are important to achieve intended effects within a 
particular setting. By identifying and scientifically demonstrating the building blocks of effective prevention programs, the 
National Cross-Site Evaluation has provided an important resource for the practice of science-based prevention. 
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